Convergence and Monotonicity Problems in an Information-Theoretic Law of Small Numbers

A version of the law of small numbers is analyzed in information-theoretic terms. Specifically, let f = {fi, i = 0, 1, . . .} be a probability mass function (pmf) on nonnegative integers with mean λ < ∞. Denote the nth convolution of f by f and denote the α-thinning of f by Tα(f). Then, as n → ∞, the entropy H(T1/n(f )) tends to H(po(λ)), where po(λ) denotes the pmf of the Poisson distribution with mean λ, and the relative entropy D(T1/n(f )|po(λ)) tends to zero, if it ever becomes finite. Moreover, α−1D(Tα(f)|po(αλ)) increases in α ∈ (0, 1), and nD (f|po(nλ)) decreases in n = 1, 2, . . .. It follows that D(T1/n(f )|po(λ)) decreases monotonically in n. Furthermore, assuming that f is ultra-log-concave (i.e., logconcave relative to the Poisson pmf), we show that H(T1/n(f )) increases monotonically in n. This is a discrete analogue of the monotonicity of entropy considered by Artstein et al. (2004). A convergence rate is also obtained. If f is either ultra-log-concave or has finite support, then D(T1/n(f )|po(λ)) = O(n), as n → ∞. In general, our results extend the parallel between the information-theoretic central limit theorem and the informationtheoretic law of small numbers explored by Kontoyiannis et al. (2005) and Harremoës et al. (2007, 2008). Ingredients in the proofs include convexity, majorization, and stochastic orders.